Printer With Embedded Retrieval and Publishing Interface 

Inventors: 
Jonathan J. Hull 

Kurt Piersol 
Jamey Graham 

Peter E. Hart 

CROSS-REFERENCES TO RELATED APPLICATIONS 

[0001] The present application is a continuation-in-part of U.S. Patent Application 
09/728,453, filed Nov 30, 2000, entitled "Techniques for Receiving Information During 
Multimedia Presentations and Communicating the Information," which is herein incorporated by 
reference in its entirety. 

[0002] The present application is a continuation-in-part of U.S. Patent Application 
09/728,560, filed Nov 30, 2000, entitled "Techniques for Capturing Information During 
Multimedia Presentations," which is herein incorporated by reference in its entirety. 
[0003] The present application claims priority under 35 U.S.C. § 1 19(e) to: 

1) U.S. Provisional patent application entitled "Printer Including One or More 
Specialized Hardware Devices" filed on September 25, 2003, having serial 
number 60/506,303, which is incorporated by reference in its entirety, and 

2) U.S. Provisional patent application entitled "Printer Including Interface and 
Specialized Information Processing Capabilities" filed on September 25, 
2003, having serial number 60/506,302, which is incorporated by reference in 
its entirety. 

[0004] The following patent applications are related to the present application and are 
incorporated by reference herein in their entirety: 

1) U.S. Patent Application No. , filed March 30, 2004, of 

Hart et al., entitled "Printer Having Embedded Functionality for Printing Time- 
Based Media," attorney docket number 20412-08340. 

2) U.S. Patent Application No. , filed March 30, 2004, of Hart 

et al., entitled " Printer User Interface," attorney docket number 20412-08455. 



1 



20412/08421/DOCS/1416586.3 



3) U.S. Patent Application No. , filed March 30, 2004, of Hart 

et al, entitled "Printer With Hardware and Software Interface for Media 
Devices," attorney docket number 20412-08383. 

4) U.S Patent: Application Serial No. 10/001,895, "(Video Paper) Paper- 
based Interface for Multimedia Information," filed November 19, 2001. 

5) Application Serial No. 10/001,849, "(Video Paper) Techniques for 
Annotating Multimedia Information," filed November 1 9, 200 1 . 

6) Application Serial No. 10/001,893, "(Video Paper) Techniques for 
Generating a Coversheet for a paper-based Interface for Multimedia Information," 
filed November 19, 2001. 

7) Application Serial No. 10/001,894, "(Video Paper) Techniques for 
Retrieving Multimedia Information Using a Paper-Based Interface," filed 
November 19, 2001. 

8) Application Serial No. 10/001,891, "(Video Paper) Paper-based 
Interface for Multimedia Information Stored by Multiple Multimedia 
Documents," filed November 1 9, 200 1 . 

9) Application Serial No. 10/175,540, "(Video Paper) Device for 
Generating a Multimedia Paper Document," filed June 18, 2002. 

10) Application Serial No. 10/645,821, "(Video Paper) Paper-Based 
Interface for Specifying Ranges CIP," filed August 20, 2003. 

BACKGROUND OF THE INVENTION 

1. Field of the Invention 

[0005] The present invention relates to document printers and, more specifically, to 
systems and methods that receive a document containing multimedia or pointers to multimedia 
and act in accordance with the received document. 

2. Description of the Background Art 

[0006] Conventional printers receive a representation of a document and then print the 

document in paper form. Generally, a conventional document will contain text and possibly 
some pictures or static graphics. For example, a printer may print a sales report or a snapshot of 
a web page. If the document is part of a larger multimedia presentation, however, conventional 
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printers do not adequately represent the complete presentation. For example a sales report might 
be part of an animated computer slide show. As another example, a web page might incorporate 
an animated picture. As a third example, a pointer or local address might point to a digital 
recording of a television program. Conventional printers can only print a captured snapshot of 
such multimedia presentations. 

[0007] Today, as more and more electrical databases and computer networks are 
interconnected, people often have multiple data systems and destinations in which to store 
information. For example, a person may receive an email containing information and/or a 
multimedia attachment that he wants to retain. The person may want to print some or all of the 
information. The person may further want to add the information to a database or to send the 
information to other people or destinations or to add the information to a web page. Currently, 
the person will need to execute several different software programs and will need to type 
multiple commands into the programs. He may also need to re-enter the information into one or 
more programs. This is not efficient and is prone to human error, since human beings 
occasionally forget to perform one of more of the tasks usually associated with a received 
document and are also prone to typographical errors. 

[0008] What is needed is a printer and method that can capture and print more of the true 
flavor of a multimedia presentation or multimedia document. 

SUMMARY OF THE INVENTION 
[0009] The present invention overcomes the deficiencies and limitations of the prior art by 
providing a system and method for a printer that can receive pointers or links to multimedia data 
and trigger an action in response. For example, one embodiment of the printer receives 
multimedia information such as a pointer to multimedia content, then retrieves the multimedia 
content and prints a document representing the multimedia content. Another embodiment further 
stores the retrieved multimedia content in a database or in local storage of the printer. Another 
embodiment of a printer gathers multimedia content that is accessible to it (e.g., via a network or 
in its local storage) and prints a summary document of the gathered multimedia content. 
[0010] The action performed by the printer preferably is based on content of the document 
received by the printer. This content can include printable and non-printable data. In certain 
embodiments, the document is analyzed by a data processing system before it is sent to be 
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printed and additional data is added to or coupled with the document. In one embodiment, an 
author or other user provides additional information, such as the name of a database, a special 
print command, or a pointer to a web page. In another embodiment, plug-in software on the data 
processing system analyzes the document and provides additional information about its content. 
In another embodiment, printer driver software on the data processing system analyzes the 
document and provides additional information about its content. In yet another embodiment, the 
printer analyzes the document after the printer receives it. In certain embodiments, the printer 
retrieves multimedia data in accordance with the document and prints a document in accordance 
with the document received. 

[0011] Certain embodiments retrieve multimedia data and store it locally or on a web site. 
This could be used to "print" a URL that points to an mp3 audio file or a video file. Because it 
might take a long time to download the data, the user would be happy if the printer takes care of 
that in the background. Subsequently, the user can view a listing of the audio and video files on 
a web page served by the printer and play selected files on demand. That web page could also 
allow the user to generate video paper for those files. This could result in the printing of a paper 
document, but it could also provide an electronic representation (e.g., pdf) for the video paper. 
The bar codes would refer to the location of the multimedia after it was retrieved by the printer. 
[0012] In other embodiments, the multimedia data itself is sent to the printer in conjunction 
with or as a part of the document. In some embodiments, the received pointers or multimedia 
data must be processed before printing can occur. Certain embodiments also store the retrieved 
multimedia data in local storage or in a database. 

BRIEF DESCRIPTION OF THE DRAWINGS 
[0013] The invention is illustrated by way of example, and not by way of limitation in the 
figures of the accompanying drawings in which like reference numerals refer to similar elements. 
[0014] Fig. 1(a) is a block diagram showing a system usable in connection with the present 
invention. 

[0015] Fig. 1(b) is a block diagram showing a system usable in connection with the present 
invention. 

[0016] Fig. 1(c) is a block diagram showing a system usable in connection with the present 
invention. 



4 



20412/08421/DOCS/1416586.3 



[0017] Fig. 1(d) is a block diagram showing a system usable in connection with the present 
invention. 

[0018] Fig. 1 (e) is a block diagram showing a system usable in connection with the present 
invention. 

[0019] Fig. 2 shows an embodiment of the present invention for printing a PDF form and 
making an entry based on content of the form into a database. 

[0020] Fig. 3(a) shows a first embodiment of the present invention for printing a paper 
document. 

[0021] Fig. 3(b) shows a second embodiment of the present invention for printing a paper 
document. 

[0022] Fig. 3(c) shows a third embodiment of the present invention for printing a paper 
document. 

[0023] Fig. 3(d) shows a fourth embodiment of the present invention for printing a paper 
document. 

[0024] Fig. 4 shows an example of interactive communication with a printer in accordance 
with the present invention. 

[0025] Fig. 5(a) is a flowchart corresponding to an embodiment of Fig. 3(a). 

[0026] Fig. 5(b) is a flowchart corresponding to an embodiment of Fig. 3(b). 

[0027] Fig. 5(c) is a flowchart corresponding to an embodiment of Fig. 3(c). 

[0028] Fig. 5(d) is a flowchart corresponding to an embodiment of Fig. 3(d). 

DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS 
[0029] A printer and associated method that can retrieve and print a representation of 
multimedia is described. For example, a printer receives a document containing or referencing 
audio and/or video multimedia data. The printer then produces a printed document in 
accordance with the received document. In this application, the term multimedia can include any 
type of multimedia content such as, but limited to, graphics (JPEGs, Gifs, etc), animation, video 
from any source and in any format, photographs, scans, audio from any source and in any format, 
and combined forms such as Macromedia Flash files, or static compound documents whose 
contents include embedded video, audio, or animation. In the following description, for 
purposes of explanation, numerous specific details are set forth in order to provide a thorough 
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understanding of the invention. It will be apparent, however, to one skilled in the art that the 
invention can be practiced without these specific details. In other instances, structures and 
devices are shown in block diagram form in order to avoid obscuring the invention. 
[0030] Reference in the specification to "one embodiment," "certain embodiments" or "an 
embodiment" means that a particular feature, structure, or characteristic described in connection 
with the embodiment is included in at least one embodiment of the invention. The appearances 
of the phrase "in one embodiment" in various places in the specification are not necessarily all 
referring to the same embodiment. 

[0031] Some portions of the detailed descriptions that follow are presented in terms of 
algorithms and symbolic representations of operations on data bits within a data processor 
memory. These algorithmic descriptions and representations are the means used by those skilled 
in the data processing arts to most effectively convey the substance of their work to others skilled 
in the art. An algorithm is here, and generally, conceived to be a self-consistent sequence of 
steps leading to a desired result. The steps are those requiring physical manipulations of physical 
quantities. Usually, though not necessarily, these quantities take the form of electrical or 
magnetic signals capable of being stored, transferred, combined, compared, and otherwise 
manipulated. It has proven convenient at times, principally for reasons of common usage, to 
refer to these signals as bits, values, elements, symbols, characters, terms, numbers, or the like. 
[0032] It should be borne in mind, however, that all of these and similar terms are to be 
associated with the appropriate physical quantities and are merely convenient labels applied to 
these quantities. Unless specifically stated otherwise as apparent from the following discussion, 
it is appreciated that throughout the description, discussions utilizing terms such as "processing" 
or "computing" or "calculating" or "determining" or "displaying" or the like, refer to the action 
and processes of a computer system, or similar electronic computing device, that manipulates 
and transforms data represented as physical (electronic) quantities within the computer system's 
registers and memories into other data similarly represented as physical quantities within the 
computer system memories or registers or other such information storage, transmission or 
display devices. 

[0033] The present invention also relates to apparatus for performing the operations herein. 
This apparatus may be specially constructed for the required purposes, or it may comprise a 
general-purpose computer selectively activated or reconfigured by a computer program stored in 
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the computer. Such a computer program may be stored in a computer readable storage medium, 
such as, but is not limited to, any type of disk including floppy disks, optical disks, CD-ROMs, 
and magnetic-optical disks, read-only memories (ROMs), random access memories (RAMs), 
EPROMs, EEPROMs, magnetic or optical cards, or any type of media suitable for storing 
electronic instructions, and each coupled to a computer system bus. 
[0034] The algorithms and displays presented herein are not inherently related to any 
particular computer or other apparatus. Various general-purpose systems may be used with 
programs in accordance with the teachings herein, or it may prove convenient to construct more 
specialized apparatus to perform the required method steps. The required structure for a variety 
of these systems will appear from the description below. In addition, the present invention is not 
described with reference to any particular programming language. It will be appreciated that a 
variety of programming languages and Page Description Languages (PDLs) may be used to 
implement the teachings of the invention as described herein. 

[0035] Moreover, the present invention is claimed below operating on or working in 
conjunction with an information system. Such an information system as claimed may be the 
entire messaging system as detailed below in the preferred embodiment or only portions of such 
a system. Thus, the present invention is capable of operating with any information system from 
those with minimal functionality to those providing all the functionality disclosed herein. 
[0036] Fig. 1(a) is a block diagram showing a system usable in connection with the present 
invention. A document having document content 105 is sent from a document source 102 to a 
printer 106. As used herein, the term "document content" refers to anything in the print stream 
sent to the printer, including both printing and non-printing data. A document can have any 
appropriate format and is any print stream or command stream that describes a page, including 
for example, PDL, ASCII text, HTML, etc. 

[0037] In the Figure, the document is sent over a network 104, such as the Internet, an 
intranet, a wireless connection, a wide area network, or the like. Printer 106 receives the 
document and performs an action based on the content of the document. One example of such an 
action is to create and print a document 110 summarizing the multimedia content pointed to by 
the document. In this case, the printer would first retrieve the content. Another example of such 
an action is to create and print a document summarizing multimedia information or content that 
is contained within the document. The printer may also perform additional actions, such as 
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entering data in a database, sending a, notification or confirmation, adding data to a web page, 
etc. In this example, printer 106 retrieves data from component 109 and stores data on 
component 108. Examples of actions performed in accordance with content of the document are 
discussed in further detail below. 

[0038] Fig. 1(b) is a block diagram showing a system usable in connection with the present 
invention. In this example, document source 102 is connected to printer 106 without a network 
connection. Similarly, printer 106 is connected to components 108 and 109 without a network 
connection. In this example, printer 106 retrieves data from component 109 and stores data on 
component 108. 

[0039] Fig. 1 (c) is a block diagram showing a system usable in connection with the present 
invention. In this example, printer 106 is connected to components 108 and 109 using a 
network connection through network 104. One or both of components 108 and 109 can be so 
connected. Network 104 can be any network, such as the Internet, an intranet, a wireless 
connection, a wide area network, or the like. 

[0040] Fig. 1(d) is a block diagram showing a system usable in connection with the present 
invention. In this example, printer 106 contains local storage 111, such as an internal memory 
and/or a database. Multimedia content pointed to by or contained in document content is 
retrieved if necessary and stored in storage 111. 

[0041] Fig. 1(e) is a block diagram showing a system usable in connection with the present 
invention. In this example, printer 106 accesses multimedia content stored in a network or stored 
in local storage 1 1 1 connected to or within the printer. Printer 106 may take this action in 
response to a command in a received document or without receipt of an explicit user command. 
After printer 106 gathers accessible multimedia content and/or information about the content, the 
printer creates and prints (or stores) a summary of the accessible multimedia data. 
[0042] It will be understood that the system configurations shown in Figs. 1(a)- 1(e) are 
examples only and are included to show some configurations usable with the present invention. 
It will be understood that other configurations are possible. For example, the connections 
between the document source and the printer and between the printer and components 108 and 
109 can both be network connections. 

[0043] Fig. 2 shows an embodiment of the present invention in which a printer 204 
receives a document 202 and produces a printed document 206. In this example, document 202 
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includes text ("Everyone should watch this TV program about successful companies:") and a 
pointer to multimedia content (http://wwwxnn.eom/l 234.wmv ). A pointer might be part of a 
text document or part of a more complex document, such as a PowerPoint presentation that the 
pointer can be extracted from. A URL is an example of a multimedia pointer. Here, the pointer 
links to a video clip on the Word Wide Web. Printer 204 detects and/or extracts the URL and 
retrieves the video clip pointed to by the URL. Other pointers might point, for example, into a 
database or indexed list of documents or into data on an intranet or other local storage medium. 
In the example, printer 204 extracts key frames 234 from the video referenced by the pointer and 
prints them in document 206, together with a pointer 236 to local storage that contains the 
retrieved video clip. In this example, the printed document also prints the text 230 that was 
originally part of document 202 and the URL itself 232. Here, for example, the printer stores the 
retrieved video clip in a cache memory 225 of the printer. Such a cache may be, for example, 
internal to the printer, local to the printer (such as a hard drive or other local storage device), or 
could be connected to the printer in any known and appropriate manner. 
[0044] In Fig. 2, the pointer 236 to local storage takes the form of a bar code 236 that is a 
part of the printed document 206. This bar code refers to a location of a cached video clip. In 
other embodiments, the bar code might refer to an original location of the clip or to a location of 
the printed key frames. 

[0045] It should be noted that scanning the bar code 236 will enable a data processing 
system (not shown) to access the local cached copy of the data even if the original copy on the 
Web is removed for some reason. Thus, for example, a human being could rapidly read through 
document 206 and many similar documents. The human could then cause a data processing 
system (such as a hand-held bar code reader) to scan the bar code and immediately begin playing 
the video clip on an associated display. This would present a way for many people to 
inexpensively look through documents representing stored video clips and then to play or copy 
the clips that they desired. Storing pointers to video clips as paper documents also has an 
advantage because paper is long lasting and does not require any software or hardware for 
viewing. The term "printed document" and "printable document" as used herein is not, however, 
limited to paper documents. A printer in accordance with the present invention could also output 
other types of outputs to summarize multimedia inputs. 
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[0046] Another embodiment embeds extra data (for example, the video clip itself) in the 
print stream (over and above what is needed to print the document). The printer 204 caches the 
data and creates a paper-based summary as appropriate. 

[0047] In the example of Fig. 2, the printer 204 may initiate a dialog with a print driver 
220. Such interactive printing is contained in certain embodiments of the present invention and 
not in others. It is discussed in more detail elsewhere in this application. 
[0048] In the example of Fig. 2, the printer 204 includes a multimedia retrieval interface. 
This interface may be implemented as hardware and/or software. Through it, the printer 
communicates with sites on the network that provide multimedia content. Such an interface can 
include, for example, an interface to a locally stored index table. When printer 204 receives a 
document, it determines whether the document it receives matches a document in the index. 
[0049] In the example of Fig. 2, printer 204 further includes multimedia analysis software, 
such as software to perform clip segmentation, key frame extraction, and image matching. In the 
example of Fig. 2, printer 204 further includes processing logic that may be implemented as 
software and/or hardware and that includes video paper generation. Video paper and audio paper 
are a paper-based interface for audio and video data. Examples of video and audio paper are 
described in the following documents, which are herein incorporated by reference in their 
entirety. 

Graham and Hull, "Video Paper: A Paper-Based Interface for Skimming and 
Watching Video," 2002 International Conference on Consumer Electronics, 
June 16-18, Los Angeles, CA. 

Graham and Hull, "A Paper-Based Interface for Video Browsing and Retrieval," 
2003 IEEE Conf. On Multimedia and Expo (ICME, July 6-9, 2003, Baltimore, 
MD. 

Hull, Erol, Graham, and Lee, "Visualizing Multimedia Content on Paper 
Documents: Components of Key Frame Selection for Video Paper," 
Proceedings of the Seventh International Conf. On Document Anlaysis and 
Recognition, ICDAR 2003. 

Graham, Erol, Hull, and Lee, "The Video Paper Multimedia Playback System," 
MM 2003, November 2-8, 2003, Berkeley, CA, ACM 1-158113-722-2/03/0011. 
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[0050] In the example of Fig. 2, printer 204 further includes a printer driver interface to 
interface with a printer driver in a video source (such as a user's data processing system). In one 
embodiment, the processing logic interacts 220 with the user through a print driver dialog and 
generates video paper. 

[0051] In the example of Fig. 2, printer 204 further includes hardware and/or software to 
perform digital media output, including a network interface to read and write media including, 
for example, DVDs, SDs, a network interface card, etc. 

[0052] In various embodiments, printer 204 communicates over a network, although 
printer 204 could also have a direct connection to the various components shown. For example, 
a database can be external or internal to printer 204. In this example, printer 204 includes 
network interface hardware and software 212 required to communicate over a network. Printer 
204 further includes Internet communication software 210 that creates and executes commands 
to communicate with the database. These commands are, for example, SQL commands, http 
commands, or similar commands, depending on the nature of database 240. 
[0053] In the example of Fig. 2, printer 204 performs additional actions besides printing 
document 206. In the example, printer 204 retrieves data from a web page or site 242 and stores 
data relating to the received document 202 in a database 240. It should be noted that other 
embodiments may additionally create web pages 246 and retrieve multimedia content or 
information from database 240. Other embodiments might store and/or retrieve data to and from 
a web site 242. Other embodiments might store and/or retrieve data from a web-publishing site 
246, such as www.documentmall.com. 

[0054] Figs. 3(a)-3(d) show further implementations of a printer in accordance with the 
present invention. It will be understood that the components of these figures can be connected 
with or without a network connection and that any connection method enabling the 
communication described below falls within the sprit and scope of the present invention. 

Application Plug-in, Simple Document Processing by Printer 
[0055] Fig. 3(a) shows a first embodiment of the present invention for printing a paper 
document 306 and optionally performing an action 308. Fig. 5(a) shows a flow chart of a method 
used by this system. In Fig. 3(a), a data processing system 302 communicates with a printer 304. 
Data processing system 302 includes an initial copy of the document to be printed 322. This 
document 322 can have any appropriate format, including but not limited to ACSII text, 
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Postscript, Microsoft Word, Adobe's PDF format, etc. (Postscript is a trademark of Adobe 
Systems Incorporated). This document optionally includes author-provided data, as discussed 
below. Data processing system 302 also includes a document rendering application 324, such as 
Acrobat Reader (available from Adobe Systems Incorporated). In this example, document 
rendering application 324 includes an application plug-in 325. Application plug-in 325, inspects 
every document printed by document rendering application 324 and preferably performs the 
following (see 504): 

a. determines whether a retrieval query should be added to the document. For example, a 
plug-in searches the document for URLs that point to video and embeds them in the PDL sent to 
the printer. As another example, the plug-in 325 can construct an explicit query, such as a 
conjunction of pairs of adjacent words in the document that could be applied to a database of 
stored PowerPoint recordings that are created by a system (that we call the Presentation 
Recorder) such as the one described in U.S. Patent Application No. 09/728,453, filed Nov 30, 
2000, and U.S. Patent Application 09/728,560, filed Nov 30, 2000, entitled "Techniques for 
Capturing Information During Multimedia Presentations," both of which are herein incorporated 
by reference in their entirety. The presentation recorder saves images of presentation slides 
together with audio and video of a person describing those presentation slides. The audio and 
video are time-stamped so that a user can retrieve and replay the multimedia associated with any 
particular slide. Each slide image is OCR'd and the results are stored in a full text index. In the 
present invention, a PowerPoint plug-in could create queries to such a full text index from each 
slide. Those queries would be embedded in the comments of the pdf file and the printer would 
apply them to the Presentation Recorders that it knows about. If it finds a matching recording, 
the printer would retrieve the multimedia, store it on the printer, and inform the user that this was 
done. The printer could also print a video paper representation for that data. In some 
embodiments, the plug-in retrieves and transmits multimedia data to the printer (e.g., audio and 
video clips are added to the PDL file. These would conventionally be classified as non-printing 
data since conventional printers cannot print audio or video). 

[0056] Data processing system 302 then sends information to the printer using a format 
such as Postscript (see 506). This information is termed "Page Description Language" (PDL) 
because it describes the appearance and characteristics of a document to be printed. In this 
example, the PDL includes data in the original document to be printed and data derived by 
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application plug-in 325 (such as data extracted by the plug-in from fields in the document or 
actual multimedia data fetched by the plug-in). It will be understood that the functionality of 
plug-in 325 could also be implemented as part of application software 324, instead of as a plug- 
in. 

[0057] The communication of data to the printer in a way that does not affect the 
appearance of the printed document above can be done by including the data as a non-printing 
comment field in the PDL file. This type of non-printing comment is currently allowed in Adobe 
Postscript or Hewlett-Packard's PCL. Often a long sequence of unlikely characters signals the 
presence of such data (see "Postscript Language Document Structuring Conventions 
Specification, Version 3.0 ," Sept. 25, 1992, 

http://partners.adobe.com/asn/developer/pdfs/tn/5001.DSC_Spec.pdf). Using Postscript as an 
example, in which comments are indicated by two percent signs at the beginning of a line, data 

fields for a URL could be indicated as: 

%%RICOH-NEPV1.0-URLITEM page=l field="Pointer" type="string" value=-"http://www.cnnxom.l234" 
[0058] Performing this processing in an application plug-in means that processing can be 
specialized for an application. For example, a PowerPoint plug-in can include commands that 
trigger the printer to search in online databases of previously recorded presentations. 
[0059] Fig. 3(a) further shows a printer 304 having document processing software 326 and 
storage 329. In this example, document processing software 326: 

a. determines whether the document contains application-derived data, and whether any 
further processing should be applied to the document (see 508). For example, the printer 
searches the received pdl file for commands and data in the comments. If the commands indicate 
that the document should be searched for links or data, the pdl file is searched for those elements. 
The printer can receive query commands to retrieve image data, audio clips, URLs, or video 
clips. Functional descriptions of such commands include but are not limited to: 

-Search this document for any URLs; 

-Search this document for specific URLs; 

-Search this document for audio clips; 

-Search this document for video clips; 

-Retrieve and analyze any URL found (or a particular URL). The analysis preferably 
includes one or more of: extracting a host name, or directory information, or inferring a file type 
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from a file name extension, or extract query data from a URL based form submission. For 
example: Extract all references to the New York Times web site, or all references to "jpg" files, 
or all requests to the-local workflow server for specific documents. 

-Extract key video frames from any URL (or particular URL); 

-Add bar codes to printed output; 

-Save data on printer's local storage 

-Publish content of any URL (or content of a particular URL) to the web; 
-Ask user for confirmation before taking action; 
-Never ask for user confirmation 

b. The printer 304 optionally processes the document to produce printer-derived data, 
e.g., a filter might be applied to postscript data that detects web URL's. In this document, 
processing by the printer is classified as simple or complex. In general, simple processing 
involves the printer executing a well-defined external command or recognizing regular 
expressions (such as URLs). Complex processing by the printer involves performing a content- 
based operation on the PDL. 

In a preferred embodiment, the printer software 326 executes simple processing. It 
inspects the comments in the PDL file and when it finds: 
%%RICOH-NEP V 1 .0-CONTENT-FILTER NO 

it determines that a content-based analysis of the PDL is not needed. Instead, printer 204 
executes a command from data processing system 302. 

c. The printer decides and performs an action in accordance with the content of the 
received document, e.g., the commands in the document might indicate that the document should 
be searched for links to video files and if such links are found, the video should be retrieved, key 
frames extracted from it, and a video paper representation constructed. A video paper layout 
may, for example, disrupt the original layout of the document. Alternately, it may be constructed 
in a layout that minimizes this disruption. For example, many URLs can be rewritten as bar 
codes, using at most space occupied by the textual form of a URL in the original document. As 
another alternative, the video paper including information not in the original document (such as 
bar codes, links, etc) can be printed as a separate document or as a cover sheet to the original 
document. 
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[0060] In general, if a retrieval command is embedded in the comments and the appropriate 
data is located in the PDL, then the retrieval command is executed by the printer, the retrieved 
data modified as specified (for example, key frames extracted from the video), and the printed 
document is modified appropriately. If a retrieval command is not embedded in the comments, 
but URLs pointing to multimedia are present (or if the multimedia content itself is present in the 
PDL), the user is prompted for whether he would like the printer to execute the retrieval of 
multimedia data. As described above, the multimedia content is then retrieved and stored on the 
printer or on an associated storage device (either a default or as indicated in a command). Then 
the paper document is printed and optional additional actions are performed as described below. 
[0061] In still other embodiments, the printer can perform various summation functions 
such as speech recognition for audio files, key frame extraction for video files, and so on. 
[0062] Key frame extraction is described, for example, in the following documents, which 
are incorporated by reference in their entirety: 

Hull, Erol, Graham, and Lee, "Visualizing Multimedia Content on Paper Documents: 
Components of Key Frame Selection for Video Paper," Proceedings of the Seventh International 
Conf. On Document Anlaysis and Recognition, ICDAR 2003. 

A. Girgensohn and J. Boreczky, "Time-Constrained Keyframe Selection Technique," 
IEEE Multimedia Systems 99, IEEE Computer Society, V. 1, 756-761, 1999. 
[0063] In summary, in the system shown in Fig. 3(a), when the user executes a print 
command, the application processes the source document and creates a page description 
language (PDL) version of the document that includes application-derived data computed by an 
application plug-in. Document processing software on the printer detects the presence of that 
information and prints the original document, along with additional information in accordance 
with the original document. The printer also decides what additional actions should be executed, 
as a function of the application-derived, and printer-derived characteristics of the document. 
[0064] The following paragraphs discuss other implementations of the present invention. 

Driver Data, Simple Document Processing by Printer 
[0065] Fig. 3(b) shows a second embodiment of the present invention for printing a paper 
document and optionally performing other actions. Fig. 5(b) shows a flow chart of a method 
used by this system. In this example, a print driver 327 provides data (such as the name of a 
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database to be updated or providing a URL of multimedia data to augment the original 
document). In one embodiment, print driver 327 creates PDL files with commands embedded in 
comments that indicate whether a document should be searched by the printer for multimedia 
links or multimedia data and the action(s) that should be performed on those links or data. The 
print driver then creates a PDL file with multimedia links or data in the comments so that no 
searching of the document need be performed by the printer. 

[0066] Performing this processing in a print driver means that the functionality can be 
delivered in a generic way, perhaps across all applications. For example, an instruction could be 
added to every document sent to the printer that tells it to search the document for URLs that 
point to video clips and to create a paper-like summary for each clip. In one implementation, at 
installation, the print driver is configured to indicate its multimedia handling capabilities. For 
example, whether it should always look for URLs, always retrieve video clips before sending a 
document, always extract key frames before sending a document, and so on. 
[0067] This implementation (and that of Fig. 3(a) and 3(d)) is useful in systems where data 
sent to the printer is commonly stored in a print queue. In such systems, the'protocol does not 
allow for user interaction with the printer. Thus, any special communication must be embedded 
within the data initially sent to the printer. Ideally, the embedded data does not affect the print 
spooler or other forwarding device, while remaining easy for the printer to detect and parse. As 
an example, the user inserts the following PostScript metadata into the print stream. Note that 
this metadata has the format of a non-printing comment. 

%%RICOH-NEPV 1 .0-URL- ACTION page=l, x=1234 y= 1856 url=http://www.cnn.com/1234 

action=Retrieve_and_summarize multimedia_storage=on_printer 
The above example indicates a URL where multimedia data is located. Document processing 
software 326 of printer 304 looks for commands within the print stream and processes them. 
Thus, in the above example, software 326 would retrieve the multimedia content at 
www.cnn.com/1 234 and print a barcode on the printed document at location x=1234;y=1856, 
where the bar code indicates a storage location of the retrieved multimedia data. Other examples 
of actions taken by printer 304 in various embodiments include one or more of the following: 
-Collect all form data from the print stream and insert it into a database; 
-Perform arbitrary calculations and sending mail messages to a known address 
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-Detect web URLs in the document and instruct a company's web proxy server to pre- 
cache those pages or store an archival copy 

-Download web URLs and print those web pages as an attachment to the document being 

printed. 

No Author-provided Data, No Application Plug-in, Complex Document Processing by Printer 
[0068] Fig. 3(c) shows a third embodiment of the present invention for printing a paper 
document and additionally performing an action. Fig. 5(c) shows a flow chart of a method used 
by this system. In such an implementation, analysis of the document is performed on the printer 
304, eliminating the need for modifications, special print drivers, or plug-ins on the data 
processing system 302. 

[0069] In one embodiment, printer 304 detects web URLs in the PDL, retrieves those 
pages, and prints them as an attachment to the original document. 

[0070] In another embodiment, the printer pre-caches web pages corresponding to URLs in 
the document under the assumption that those web pages will soon be accessed by the user 
printing the document with their URLs. Alternatively, the printer 304 retrieves the web pages 
and stores them in a database. 

[0071] In another embodiment, the printer retrieves the multimedia content for some or all 
URLs in a document, creates a summary of their contents (URL, title, author, date printed, 
number of pages, thumbnail for a web page, short web page text summary, key frames from a 
video clip, etc) and prints that summary, either embedded in the original document or as an 
addendum. 

[0072] The above embodiments provide examples of applying a calculation or analysis to 
the document received by the printer 304 and performing an action (besides printing) based on 
the calculation or analysis. 

Author-provided Data, No Application Plug-in, Simple Document Processing by Printer 
[0073] Fig. 3(d) shows a fourth embodiment of the present invention for printing a paper 
document and optionally performing other actions. Fig. 5(d) shows a flow chart of a method 
used by this system. In this example, the user provides data (such as the name of a database to 
be updated or providing a URL of multimedia data to augment the original document). This 
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user-provided data can be provided with or without interaction by the user with the printer. In 
this example, the author includes a command in a document to be printed by adding the 
command as a non-printing comment field in the document. This type of non-printing comment 
is currently allowed in, for example, Adobe Postscript and Hewlett-Packard's PCL printer 
language. 

Additional Embodiments 
[0074] It will be understood by persons of ordinary skill in the art that the processing 
described above can be performed at various locations within the system. For example, the 
processing described above in connection with "complex document processing" performed by 
the printer 304 could also be performed, either totally or partially by data processing system 302. 
Similarly, processing described as being performed by software on data processing system 302 
could be performed by document processing software in printer 304. Similarly, processing could 
be performed both by the data processing system/document source and by the printer. 
[0075] One advantage of using plug-ins in data processing system 302 is that they are easy 
to add to existing software. In contrast, one advantage of performing most or all processing on 
the printer side is that printers are easier to maintain and service than user computers. In 
addition, if a large enterprise is going to purchase special processing software, it is less 
expensive to purchase software that runs on a small number of printers as opposed to software 
that runs on each user machine. This is even more important if special purpose hardware (such as 
a powerful signal processor) is needed to perform the function, because of the expense and 
difficulty of such installations. Moreover, the work environment of printers is usually better 
monitored and it is physically easier to maintain security over printers that contain sensitive 
analysis programs (such as programs that monitor compliance with licensing or seat 
requirements or with security requirements) than if this analysis software was located in users' 
computers. 

[0076] The following paragraphs contain additional examples of "complex processing" by 
a printer. The printer recognizes a PDL file directly and analyzes the file to detect certain known 
fields. As stated above, these embodiments could also be implemented on data processor 302. 

- Printer with Embedded Multimedia Retrieval : When a user prints, for example, 
PowerPoint slides, the printer searches a database of previously recorded audio/video 



18 



204 1 2/0842 l/DOCS/14 1 6586.3 



presentations including slides. The printer matches images extracted from the PowerPoint slides 
with images saved by the presentation recorder. A video paper printout is created and printed 
that includes bar codes providing access to the previously saved multimedia presentations, along 
with the slides that were originally sent to be printed. 

- Printer With Embedded Video Clip Segmentation and Web Publishing : A user "prints" a 
video recording that is segmented into clips by the printer. The printer displays the results to the 
user in a print dialog box. The user can then choose to publish clips on the web. The clips are 
either made available on a web server resident in the printer or copied to another web repository 
(for example, www. documentmal 1 .com) . The printer also produces a paper record of this 
transaction that includes the web address where the clips can be found. Other embodiments 
choose clips interactively, via a print driver or plug-in, or by using a default defined within the 
printer. 

- Printer with Embedded Audio Clip Segmentation and Web Publishing : A user "prints" 
an audio recording that is segmented into clips by the printer. The printer displays the results to 
the user in a print dialog box. The user can then choose to publish clips on the web. The clips 
are either made available on a web server resident in the printer or copied to another web 
repository (for example, www.documentmall.com ). The printer also produces a paper record of 
this transaction that includes the web address where the clips can be found. Other embodiments 
can choose clips interactively, via a print driver or plug-in, or by using a default defined within 
the printer. 

[0077] Other embodiments may automatically choose clips or samples based on identifying 
a face in the document and finding audio clips of that person speaking. Other embodiments may 
print an automatic transcript of the audio clips. 

- Printer with Embedded WWW search engine : The user enters a World Wide Web 
search query on a print dialog box. The printer does the search and displays the results. The 
results are formatted to take advantage of the printer's special capabilities (e.g., its ability to print 
in color and on legal size paper). In this embodiment, the printer includes a network interface 
and its own web crawler, which yield a very fast search. In a related embodiment, the printer 
receives a file of URLs, such as a printout of a favorites file from a browser and prints a 
summary of the contents, where the summary includes retrieved multimedia data. 
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- Printer with Embedded Video Clip Retrieval and Key frame Selection : The user 
"prints" a web document that contains references to video clips. The video clips are 
automatically downloaded and key frames are chosen from each one. A set of key frames from 
each clip (e.g., 5 key frames per clip) are printed near each link in the document. This 
embodiment gives the user an idea about the content of each clip before deciding whether it 
should be retrieved and played. 

- Printer with Embedded Image Search Engine : The user "prints" a document containing 
an image such as a logo and the printer executes a Web search for similar images of different 
qualities. Those images are displayed to the user in a print dialog box. An example usage would 
be someone printing a letter with a poor quality company logo on it. The printer would 
automatically find a better logo, thus improving the overall appearance of the document and the 
user's satisfaction with the printer. In certain embodiments, the printer would get the user's 
approval before making the substitution. 

- Printer with Embedded Weather Map Retrieval : A user issues a "print weather" 
command to the printer (see, for example, Fig. 3(d)). The printer would automatically contact 
the National Weather Service (NWS) and download a weather map as well as a text summary for 
current and forecast conditions. The characteristics of the data are determined from a print 
dialog interface. The printer could periodically query a web site such as 
http://www.wrh.noaa.gov/Monterey/ and download the latest weather maps and textual forecasts 
such as http://www.wrh.noaa.gov/cgi- 
birb^hq/TotalForecast.csh^ 

When a user prints some other document to this printer, this previously downloaded weather data 
could be displayed in the print dialog box and the user could either just view it right there or 
print some selection from it as an addendum to the current document or as a separate document 
(i.e., printing the current document is not required). Alternatively, instead of using the print 
dialog box, similar functionality could be provided with a web browser interface. Another 
alternative system configuration would contact the NWS web site when the user issues the print 
command. The advantage is that the latest information would be obtained. A disadvantage 
might be the time required to wait for the retrieval to be performed. Also, the web site might be 
down at the particular moment when the query is issued. 
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^ Printer with Embedded Aerial Image Retrieval : A user enters a street address in a print 
dialog box and the printer retrieves an aerial image from an external server (for example, 
Mapquest.com) and prints the image at a user-prescribed resolution with a user-specified color 
scheme. Alternatively, the printer maintains or has access to a local database of aerial images. 
Use of a local database reduces access times and maintains privacy of the user. Use of a local 
database keeps outside persons from knowing what has been searched for or printed. 

- Printer with Embedded Aerial Image Recognition and Highlighting : A user "prints" an 
aerial image and the printer recognizes specified features in those images, such as automobiles or 
houses. Those features are highlighted in the output image. An alternative configuration 
contains a method for periodically downloading aerial images for a specified set of locations and 
applying the aerial images recognition techniques repeatedly. This provides a way for the printer 
to notice differences over a period of time. Note that placing the recognition software in the 
printer increases the security of the software. This is important is the recognition software is 
secret or maintained with high security. It also makes it easier to execute the recognition 
software on a high-performance processor since there are usually fewer printers than user 
machines. 

Interactive Communication with a Printer 
[0078] Fig. 4 shows an example of interactive communication with a printer in accordance 
with the present invention. 

[0079] In general, conventional printer drivers in modern operating systems are not 
designed to facilitate interactive information gathering. Because the print job can be redirected 
to another printer, or the printing protocol does not allow such interactive sessions, the operating 
system does not encourage interaction with the user. Once initial printer settings are captured, 
further interactions are generally not allowed in conventional printers. One approach to this 
problem is to embed metadata into the print stream itself, as noted above. However, it is possible 
that the printer could need to ask the user for more information, in response to computations 
made from the data supplied by the user. In addition, the printer itself might delegate some tasks 
to other application servers, which might in turn need more information from the user. So-called 
"Web services" or "grid computing" systems are examples of the sort of application server that 
the printer might trigger. 
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[0080] In order to allow this interaction, without modifying printer driver architecture of 
the underlying operating system, an extra mechanism, such as the one shown in Fig. 4, is 
constructed. A "UI Listener," program 454 listens to a network socket, accepts requests for 
information 408, interacts with a user to obtain such data, and then sends the data back to the 
requester. 

[0081] Once a print request 402 is sent by user 450 and notification requested from the UI 
listener 404, the print job is sent by application 452. Here, the print job contains embedded 
information including the network address of the UI listener, authentication information, and the 
latest time that the client will be listening for requests. 

[0082] If the printer requires additional information of confirmation, it sends a request 408, 
which is detected by the UI listener, which displays a dialog box to obtain input from the user 
410. An example of such as request might be a request for a password or user confirmation code 
that the user must enter to access a database 458. Alternatively, the printer asks for additional 
information based on computation, such as noting the absence of a required field and asking for a 
value to be entered before printing. The user's input is included in a reply 412 sent to the printer. 
If the reply does not satisfy the printer it may ask for additional information (not shown). If the 
reply does satisfy the printer, it takes a next step. This step might be to perform an action such as 
sending an email (not shown). The next step might also be sending a request for information 414 
to an application server (such as a database) 458. In this example, application server 458 also 
sends a request for information 416, which is detected by the UI listener. The user is prompted 
418 and his response forwarded to the application server 420. In this example, a reply is then 
sent from the application server 458 to the printer 456. It will be understood that a particular 
embodiment may include either or none or requests 408 and 416 without departing from the 
spirit of the present invention. 

[0083] A program such as that shown in Fig. 4 may have a fixed set of possible 
interactions, or may accept a flexible command syntax that allows the requester to display many 
different requests. An example of such a command syntax would be the standard web browser's 
ability to display HTML forms. These forms are generated by a remote server, and displayed by 
the browser, which then returns results to the server. In this embodiment, however, the UI 
listener is different from a browser in that a user does not generate the initial request to see a 
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form. Instead, the remote machine generates this request. In the described embodiment, the UI 
listener is a server, not a client. 

[0084] Because network transactions of this type are prone to many complex error 
conditions, a system of timeouts would be necessary to assure robust operation. Normally, each 
message sent across a network either expects a reply or is a one-way message. Messages which 
expect replies generally have a timeout, a limited period of time during which it is acceptable for 
the reply to arrive. In this embodiment, embedded metadata would include metadata about a UI 
listener that will accept requests for further information. Such metadata preferably includes at 
least a network address, port number, and a timeout period. It might also include authentication 
information, designed to prevent malicious attempts to elicit information from the user. Because 
the user cannot tell whether the request is coming from a printer, a delegated server, or a 
malicious agent, prudence suggests strong authentication by the UI listener. If the printer or a 
delegated application server wishes more information, it can use the above noted information to 
request that the UI listener ask a user for the needed information. 

[0085] While the present invention has been described with reference to certain preferred 
embodiments, those skilled in the art will recognize that various modifications may be provided. 
For example, certain embodiments employ multiple application servers, acting in cooperation to 
perform a function or request. Any of the above functions or requests can be sent across a 
network, or using local cables such as IEEE 1394, Universal Serial Bus, or wireless networks 
such as IEEE 802.1 1 or IEEE 802.15 networks, in any combination. Variations upon and 
modifications to the preferred embodiments are provided for by the present invention, which is 
limited only by the following claims. 
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